feat(writer): Add clustered and fanout writer #1735
Conversation
| /// Build the iceberg writer. | ||
| async fn build(self) -> Result<Self::R>; | ||
| /// Build the iceberg writer for an optional partition key. | ||
| async fn build_with_partition(self, partition_key: Option<PartitionKey>) -> Result<Self::R>; |
There was a problem hiding this comment.
This is a breaking change. I believe this is necessary because:
-
IcebergWriteris supposed to generateDataFilethat always hold a partition value according to iceberg spec. -
The existing code store partition value in the builder directly, making
builder.clone()useless:
let builder = IcebergWriterBuilder::new(partition_A);
let writer_A = builder.build();
... // write to partition A
// done with partition A and now we need to write to partition B
// this is wrong because partition value A is still stored in the builder
let writer_B = builder.clone().build() An alternative is to add a new method clone_with_partition() but that would also be a breaking change and it's less clean compared to build_with_partition()
|
|
||
| /// A writer that writes data to a single partition at a time. | ||
| #[derive(Clone)] | ||
| pub struct ClusteredDataWriter<B: IcebergWriterBuilder> { |
There was a problem hiding this comment.
ClusteredDataWriter and FanoutDataWriter are supposed to only work with DefaultInput and DefaultOutput.
I tried including generic i/o few weeks ago and remember there were tons of tricky nuances and decided to just go for the default IO type for now. For other IO type(e.g. PositionalDeleteInput), we will need add another implementation later.
Also maybe we should name these DefaultClustered/FanoutWriter to avoid confusion since they can also write equality deletes?
1 participant
Which issue does this PR close?
What changes are included in this PR?
New:
partitioningmodule withPartitioningWritertraitClusteredDataWriter: Optimized for pre-sorted data, requires writing in partition orderFanoutDataWriter: Flexible writer that can handle data from any partition at any timeModification:
DataFileWriterBuilderto support dynamic partition assignmentAre these changes tested?
Added unit tests